{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/moj-analytical-services/splink/blob/master/docs/demos/examples/duckdb/pairwise_labels.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Estimating m from a sample of pairwise labels\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "In this example, we estimate the m probabilities of the model from a table containing pairwise record comparisons which we know are 'true' matches. For example, these may be the result of work by a clerical team who have manually labelled a sample of matches.\n",
        "\n",
        "The table must be in the following format:\n",
        "\n",
        "| source_dataset_l | unique_id_l | source_dataset_r | unique_id_r |\n",
        "| ---------------- | ----------- | ---------------- | ----------- |\n",
        "| df_1             | 1           | df_2             | 2           |\n",
        "| df_1             | 1           | df_2             | 3           |\n",
        "\n",
        "It is assumed that every record in the table represents a certain match.\n",
        "\n",
        "Note that the column names above are the defaults. They should correspond to the values you've set for [`unique_id_column_name`](https://moj-analytical-services.github.io/splink/settings_dict_guide.html#unique_id_column_name) and [`source_dataset_column_name`](https://moj-analytical-services.github.io/splink/settings_dict_guide.html#source_dataset_column_name), if you've chosen custom values.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:22.461384Z",
          "iopub.status.busy": "2024-06-07T09:20:22.461075Z",
          "iopub.status.idle": "2024-06-07T09:20:22.466162Z",
          "shell.execute_reply": "2024-06-07T09:20:22.465529Z"
        },
        "tags": [
          "hide_input"
        ]
      },
      "outputs": [],
      "source": [
        "# Uncomment and run this cell if you're running in Google Colab.\n",
        "# !pip install splink"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:22.470034Z",
          "iopub.status.busy": "2024-06-07T09:20:22.469740Z",
          "iopub.status.idle": "2024-06-07T09:20:24.546756Z",
          "shell.execute_reply": "2024-06-07T09:20:24.546033Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>unique_id_l</th>\n",
              "      <th>source_dataset_l</th>\n",
              "      <th>unique_id_r</th>\n",
              "      <th>source_dataset_r</th>\n",
              "      <th>clerical_match_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>2</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>3</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>1</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>2</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>1</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>3</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3171</th>\n",
              "      <td>994</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>996</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3172</th>\n",
              "      <td>995</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>996</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3173</th>\n",
              "      <td>997</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>998</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3174</th>\n",
              "      <td>997</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>999</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3175</th>\n",
              "      <td>998</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>999</td>\n",
              "      <td>fake_1000</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2031 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      unique_id_l source_dataset_l  unique_id_r source_dataset_r  \\\n",
              "0               0        fake_1000            1        fake_1000   \n",
              "1               0        fake_1000            2        fake_1000   \n",
              "2               0        fake_1000            3        fake_1000   \n",
              "49              1        fake_1000            2        fake_1000   \n",
              "50              1        fake_1000            3        fake_1000   \n",
              "...           ...              ...          ...              ...   \n",
              "3171          994        fake_1000          996        fake_1000   \n",
              "3172          995        fake_1000          996        fake_1000   \n",
              "3173          997        fake_1000          998        fake_1000   \n",
              "3174          997        fake_1000          999        fake_1000   \n",
              "3175          998        fake_1000          999        fake_1000   \n",
              "\n",
              "      clerical_match_score  \n",
              "0                      1.0  \n",
              "1                      1.0  \n",
              "2                      1.0  \n",
              "49                     1.0  \n",
              "50                     1.0  \n",
              "...                    ...  \n",
              "3171                   1.0  \n",
              "3172                   1.0  \n",
              "3173                   1.0  \n",
              "3174                   1.0  \n",
              "3175                   1.0  \n",
              "\n",
              "[2031 rows x 5 columns]"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from splink.datasets import splink_dataset_labels\n",
        "\n",
        "pairwise_labels = splink_dataset_labels.fake_1000_labels\n",
        "\n",
        "# Choose labels indicating a match\n",
        "pairwise_labels = pairwise_labels[pairwise_labels[\"clerical_match_score\"] == 1]\n",
        "pairwise_labels"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now proceed to estimate the Fellegi Sunter model:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:24.588843Z",
          "iopub.status.busy": "2024-06-07T09:20:24.588530Z",
          "iopub.status.idle": "2024-06-07T09:20:24.602952Z",
          "shell.execute_reply": "2024-06-07T09:20:24.602047Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>unique_id</th>\n",
              "      <th>first_name</th>\n",
              "      <th>surname</th>\n",
              "      <th>dob</th>\n",
              "      <th>city</th>\n",
              "      <th>email</th>\n",
              "      <th>cluster</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>Robert</td>\n",
              "      <td>Alan</td>\n",
              "      <td>1971-06-24</td>\n",
              "      <td>NaN</td>\n",
              "      <td>robert255@smith.net</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>Robert</td>\n",
              "      <td>Allen</td>\n",
              "      <td>1971-05-24</td>\n",
              "      <td>NaN</td>\n",
              "      <td>roberta25@smith.net</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   unique_id first_name surname         dob city                email  cluster\n",
              "0          0     Robert    Alan  1971-06-24  NaN  robert255@smith.net        0\n",
              "1          1     Robert   Allen  1971-05-24  NaN  roberta25@smith.net        0"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from splink import splink_datasets\n",
        "\n",
        "df = splink_datasets.fake_1000\n",
        "df.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:24.607247Z",
          "iopub.status.busy": "2024-06-07T09:20:24.606935Z",
          "iopub.status.idle": "2024-06-07T09:20:24.711369Z",
          "shell.execute_reply": "2024-06-07T09:20:24.710531Z"
        }
      },
      "outputs": [],
      "source": [
        "import splink.comparison_library as cl\n",
        "from splink import DuckDBAPI, Linker, SettingsCreator, block_on\n",
        "\n",
        "settings = SettingsCreator(\n",
        "    link_type=\"dedupe_only\",\n",
        "    blocking_rules_to_generate_predictions=[\n",
        "        block_on(\"first_name\"),\n",
        "        block_on(\"surname\"),\n",
        "    ],\n",
        "    comparisons=[\n",
        "        cl.NameComparison(\"first_name\"),\n",
        "        cl.NameComparison(\"surname\"),\n",
        "        cl.DateOfBirthComparison(\n",
        "            \"dob\",\n",
        "            input_is_string=True,\n",
        "        ),\n",
        "        cl.ExactMatch(\"city\").configure(term_frequency_adjustments=True),\n",
        "        cl.EmailComparison(\"email\"),\n",
        "    ],\n",
        "    retain_intermediate_calculation_columns=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:24.715481Z",
          "iopub.status.busy": "2024-06-07T09:20:24.715162Z",
          "iopub.status.idle": "2024-06-07T09:20:25.100461Z",
          "shell.execute_reply": "2024-06-07T09:20:25.099741Z"
        }
      },
      "outputs": [],
      "source": [
        "linker = Linker(df, settings, db_api=DuckDBAPI(), set_up_basic_logging=False)\n",
        "deterministic_rules = [\n",
        "    \"l.first_name = r.first_name and levenshtein(r.dob, l.dob) <= 1\",\n",
        "    \"l.surname = r.surname and levenshtein(r.dob, l.dob) <= 1\",\n",
        "    \"l.first_name = r.first_name and levenshtein(r.surname, l.surname) <= 2\",\n",
        "    \"l.email = r.email\",\n",
        "]\n",
        "\n",
        "linker.training.estimate_probability_two_random_records_match(deterministic_rules, recall=0.7)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:25.104541Z",
          "iopub.status.busy": "2024-06-07T09:20:25.104116Z",
          "iopub.status.idle": "2024-06-07T09:20:26.866642Z",
          "shell.execute_reply": "2024-06-07T09:20:26.866007Z"
        }
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "You are using the default value for `max_pairs`, which may be too small and thus lead to inaccurate estimates for your model's u-parameters. Consider increasing to 1e8 or 1e9, which will result in more accurate estimates, but with a longer run time.\n"
          ]
        }
      ],
      "source": [
        "linker.training.estimate_u_using_random_sampling(max_pairs=1e6)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:26.871363Z",
          "iopub.status.busy": "2024-06-07T09:20:26.871016Z",
          "iopub.status.idle": "2024-06-07T09:20:27.051023Z",
          "shell.execute_reply": "2024-06-07T09:20:27.050407Z"
        }
      },
      "outputs": [],
      "source": [
        "# Register the pairwise labels table with the database, and then use it to estimate the m values\n",
        "labels_df = linker.table_management.register_labels_table(pairwise_labels, overwrite=True)\n",
        "linker.training.estimate_m_from_pairwise_labels(labels_df)\n",
        "\n",
        "\n",
        "# If the labels table already existing in the dataset you could run\n",
        "# linker.training.estimate_m_from_pairwise_labels(\"labels_tablename_here\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:27.054211Z",
          "iopub.status.busy": "2024-06-07T09:20:27.053972Z",
          "iopub.status.idle": "2024-06-07T09:20:27.489093Z",
          "shell.execute_reply": "2024-06-07T09:20:27.488564Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<EMTrainingSession, blocking on l.\"first_name\" = r.\"first_name\", deactivating comparisons first_name>"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "training_blocking_rule = block_on(\"first_name\")\n",
        "linker.training.estimate_parameters_using_expectation_maximisation(training_blocking_rule)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:27.492742Z",
          "iopub.status.busy": "2024-06-07T09:20:27.492510Z",
          "iopub.status.idle": "2024-06-07T09:20:27.624619Z",
          "shell.execute_reply": "2024-06-07T09:20:27.624114Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "<style>\n",
              "  #altair-viz-9d310b56c70042a58417e5a15b5c4297.vega-embed {\n",
              "    width: 100%;\n",
              "    display: flex;\n",
              "  }\n",
              "\n",
              "  #altair-viz-9d310b56c70042a58417e5a15b5c4297.vega-embed details,\n",
              "  #altair-viz-9d310b56c70042a58417e5a15b5c4297.vega-embed details summary {\n",
              "    position: relative;\n",
              "  }\n",
              "</style>\n",
              "<div id=\"altair-viz-9d310b56c70042a58417e5a15b5c4297\"></div>\n",
              "<script type=\"text/javascript\">\n",
              "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
              "  (function(spec, embedOpt){\n",
              "    let outputDiv = document.currentScript.previousElementSibling;\n",
              "    if (outputDiv.id !== \"altair-viz-9d310b56c70042a58417e5a15b5c4297\") {\n",
              "      outputDiv = document.getElementById(\"altair-viz-9d310b56c70042a58417e5a15b5c4297\");\n",
              "    }\n",
              "    const paths = {\n",
              "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
              "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
              "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.17.0?noext\",\n",
              "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
              "    };\n",
              "\n",
              "    function maybeLoadScript(lib, version) {\n",
              "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
              "      return (VEGA_DEBUG[key] == version) ?\n",
              "        Promise.resolve(paths[lib]) :\n",
              "        new Promise(function(resolve, reject) {\n",
              "          var s = document.createElement('script');\n",
              "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "          s.async = true;\n",
              "          s.onload = () => {\n",
              "            VEGA_DEBUG[key] = version;\n",
              "            return resolve(paths[lib]);\n",
              "          };\n",
              "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
              "          s.src = paths[lib];\n",
              "        });\n",
              "    }\n",
              "\n",
              "    function showError(err) {\n",
              "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
              "      throw err;\n",
              "    }\n",
              "\n",
              "    function displayChart(vegaEmbed) {\n",
              "      vegaEmbed(outputDiv, spec, embedOpt)\n",
              "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
              "    }\n",
              "\n",
              "    if(typeof define === \"function\" && define.amd) {\n",
              "      requirejs.config({paths});\n",
              "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
              "    } else {\n",
              "      maybeLoadScript(\"vega\", \"5\")\n",
              "        .then(() => maybeLoadScript(\"vega-lite\", \"5.17.0\"))\n",
              "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
              "        .catch(showError)\n",
              "        .then(() => displayChart(vegaEmbed));\n",
              "    }\n",
              "  })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}, \"title\": {\"anchor\": \"middle\", \"fontSize\": 18, \"subtitleFontSize\": 14}}, \"data\": {\"name\": \"data-ed1f09f27dd97222de32cebf09987fb4\"}, \"mark\": {\"type\": \"point\", \"filled\": false, \"opacity\": 0.7, \"size\": 100}, \"encoding\": {\"color\": {\"field\": \"estimate_description\", \"type\": \"nominal\"}, \"column\": {\"align\": \"each\", \"field\": \"col_header\", \"header\": {\"labelFontSize\": 14, \"labelFontWeight\": \"bold\"}, \"title\": null, \"type\": \"nominal\"}, \"row\": {\"align\": \"each\", \"field\": \"comparison_name\", \"header\": {\"labelAlign\": \"left\", \"labelAnchor\": \"middle\", \"labelAngle\": 0, \"labelFontSize\": 12, \"labelFontWeight\": \"bold\"}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"title\": null, \"type\": \"nominal\"}, \"shape\": {\"field\": \"estimate_description\", \"scale\": {\"range\": [\"circle\", \"square\", \"triangle\", \"diamond\"]}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"type\": \"nominal\"}, {\"field\": \"estimate_description\", \"type\": \"nominal\"}, {\"field\": \"estimated_probability\", \"type\": \"quantitative\"}], \"x\": {\"axis\": {\"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}}, \"field\": \"estimated_probability_as_log_odds\", \"title\": \"Estimated probability as log odds\", \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"grid\": true, \"title\": null}, \"field\": \"comparison_level_label\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"params\": [{\"name\": \"mouse_zoom\", \"select\": {\"type\": \"interval\", \"encodings\": [\"x\"]}, \"bind\": \"scales\"}], \"resolve\": {\"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Comparison of parameter estimates across training sessions\", \"subtitle\": \"Use mousewheel to zoom\"}, \"transform\": [{\"calculate\": \"datum.m_or_u + '-probability (as log odds)'\", \"as\": \"col_header\"}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-ed1f09f27dd97222de32cebf09987fb4\": [{\"m_or_u\": \"m\", \"estimated_probability\": 0.4953998584571833, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -0.026547154611559317, \"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"comparison_level_label\": \"Exact match on first_name\", \"comparison_vector_value\": 4, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.16772823779193205, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -2.3109290959381057, \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.92\", \"comparison_level_label\": \"Jaro-Winkler distance of first_name >= 0.92\", \"comparison_vector_value\": 3, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.050955414012738856, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -4.219168520462161, \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of first_name >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.08138711960368011, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -3.496584617054763, \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.7\", \"comparison_level_label\": \"Jaro-Winkler distance of first_name >= 0.7\", \"comparison_vector_value\": 1, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.20452937013446568, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -1.959500637720247, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"first_name\", \"comparison_sort_order\": 0}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.4485294117647059, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -0.2980813529329946, \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"comparison_level_label\": \"Exact match on surname\", \"comparison_vector_value\": 4, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.6257815226019211, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": 0.7417782660774624, \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"comparison_level_label\": \"Exact match on surname\", \"comparison_vector_value\": 4, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.21544117647058825, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -1.864587606402177, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.92\", \"comparison_vector_value\": 3, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.2275262574980547, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -1.7634528464982868, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.92\", \"comparison_vector_value\": 3, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.04044117647058824, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -4.568474377932588, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.03662391383195619, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -4.7172412336516025, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.08161764705882353, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -3.492141895257851, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.7\", \"comparison_vector_value\": 1, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.09678319985317106, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -3.2222437773864243, \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"comparison_level_label\": \"Jaro-Winkler distance of surname >= 0.7\", \"comparison_vector_value\": 1, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.2139705882352941, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -1.8771707948122067, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.013285106214896938, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -6.214751618166397, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"surname\", \"comparison_sort_order\": 1}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.42146725750861647, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -0.4569780550512149, \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"comparison_level_label\": \"Exact match on dob\", \"comparison_vector_value\": 5, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.3934524875218463, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -0.6244312304143554, \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"comparison_level_label\": \"Exact match on dob\", \"comparison_vector_value\": 5, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.08862629246676514, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -3.3622360835408993, \"sql_condition\": \"damerau_levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"comparison_level_label\": \"Levenshtein distance of dob <= 1\", \"comparison_vector_value\": 4, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.12854217781802071, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -2.7611890203595673, \"sql_condition\": \"damerau_levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"comparison_level_label\": \"Levenshtein distance of dob <= 1\", \"comparison_vector_value\": 4, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.09699655342195962, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -3.218726104138678, \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 2629800.0\", \"comparison_level_label\": \"Abs difference of 'transformed dob <= 1 month'\", \"comparison_vector_value\": 3, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.06508162067908886, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -3.8445183367349016, \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 2629800.0\", \"comparison_level_label\": \"Abs difference of 'transformed dob <= 1 month'\", \"comparison_vector_value\": 3, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.11422944362383063, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -2.9549984761239103, \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 31557600.0\", \"comparison_level_label\": \"Abs difference of 'transformed dob <= 1 year'\", \"comparison_vector_value\": 2, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.13671959131132042, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -2.65860925873792, \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 31557600.0\", \"comparison_level_label\": \"Abs difference of 'transformed dob <= 1 year'\", \"comparison_vector_value\": 2, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.23830625307730183, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -1.6763942441744413, \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 315576000.0\", \"comparison_level_label\": \"Abs difference of 'transformed dob <= 10 year'\", \"comparison_vector_value\": 1, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.22502621697661637, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -1.7840544112329517, \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 315576000.0\", \"comparison_level_label\": \"Abs difference of 'transformed dob <= 10 year'\", \"comparison_vector_value\": 1, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.04037419990152634, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -4.570966370639806, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.05117790569310731, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -4.2125445893251285, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"dob\", \"comparison_sort_order\": 2}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5661492978566149, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": 0.38398388881490503, \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"comparison_level_label\": \"Exact match on city\", \"comparison_vector_value\": 1, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5628130218653614, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": 0.3644052786018151, \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"comparison_level_label\": \"Exact match on city\", \"comparison_vector_value\": 1, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.43385070214338506, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -0.3839838888149049, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.4371869781346386, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -0.36440527860181515, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"city\", \"comparison_sort_order\": 3}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5340642129992169, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": 0.1968820708288665, \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"comparison_level_label\": \"Exact match on email\", \"comparison_vector_value\": 4, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.5707272045221725, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": 0.4109065857204091, \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"comparison_level_label\": \"Exact match on email\", \"comparison_vector_value\": 4, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.23492560689115113, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -1.7033960614725043, \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"comparison_level_label\": \"Exact match on username\", \"comparison_vector_value\": 3, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.2094073550927045, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -1.9166224179708937, \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"comparison_level_label\": \"Exact match on username\", \"comparison_vector_value\": 3, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.22239624119028975, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -1.8059027880236918, \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of email >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.19001135184876095, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -2.091816076956007, \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler distance of email >= 0.88\", \"comparison_vector_value\": 2, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.0015661707126076742, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -9.31628153174622, \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler >0.88 on username\", \"comparison_vector_value\": 1, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.019731832581364434, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -5.634579617838549, \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"comparison_level_label\": \"Jaro-Winkler >0.88 on username\", \"comparison_vector_value\": 1, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.007047768206734534, \"estimate_description\": \"estimate m from pairwise labels\", \"estimated_probability_as_log_odds\": -7.1384140286970945, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}, {\"m_or_u\": \"m\", \"estimated_probability\": 0.010122255954997663, \"estimate_description\": \"EM, blocked on: l.\\\"first_name\\\" = r.\\\"first_name\\\"\", \"estimated_probability_as_log_odds\": -6.6116475889964015, \"sql_condition\": \"ELSE\", \"comparison_level_label\": \"All other comparisons\", \"comparison_vector_value\": 0, \"comparison_name\": \"email\", \"comparison_sort_order\": 4}]}}, {\"mode\": \"vega-lite\"});\n",
              "</script>"
            ],
            "text/plain": [
              "alt.Chart(...)"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "linker.visualisations.parameter_estimate_comparisons_chart()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-07T09:20:27.628602Z",
          "iopub.status.busy": "2024-06-07T09:20:27.628256Z",
          "iopub.status.idle": "2024-06-07T09:20:27.933374Z",
          "shell.execute_reply": "2024-06-07T09:20:27.932702Z"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "<style>\n",
              "  #altair-viz-92e7ea2c6f0648648ac3ebb22c154bfb.vega-embed {\n",
              "    width: 100%;\n",
              "    display: flex;\n",
              "  }\n",
              "\n",
              "  #altair-viz-92e7ea2c6f0648648ac3ebb22c154bfb.vega-embed details,\n",
              "  #altair-viz-92e7ea2c6f0648648ac3ebb22c154bfb.vega-embed details summary {\n",
              "    position: relative;\n",
              "  }\n",
              "</style>\n",
              "<div id=\"altair-viz-92e7ea2c6f0648648ac3ebb22c154bfb\"></div>\n",
              "<script type=\"text/javascript\">\n",
              "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
              "  (function(spec, embedOpt){\n",
              "    let outputDiv = document.currentScript.previousElementSibling;\n",
              "    if (outputDiv.id !== \"altair-viz-92e7ea2c6f0648648ac3ebb22c154bfb\") {\n",
              "      outputDiv = document.getElementById(\"altair-viz-92e7ea2c6f0648648ac3ebb22c154bfb\");\n",
              "    }\n",
              "    const paths = {\n",
              "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
              "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
              "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.17.0?noext\",\n",
              "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
              "    };\n",
              "\n",
              "    function maybeLoadScript(lib, version) {\n",
              "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
              "      return (VEGA_DEBUG[key] == version) ?\n",
              "        Promise.resolve(paths[lib]) :\n",
              "        new Promise(function(resolve, reject) {\n",
              "          var s = document.createElement('script');\n",
              "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "          s.async = true;\n",
              "          s.onload = () => {\n",
              "            VEGA_DEBUG[key] = version;\n",
              "            return resolve(paths[lib]);\n",
              "          };\n",
              "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
              "          s.src = paths[lib];\n",
              "        });\n",
              "    }\n",
              "\n",
              "    function showError(err) {\n",
              "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
              "      throw err;\n",
              "    }\n",
              "\n",
              "    function displayChart(vegaEmbed) {\n",
              "      vegaEmbed(outputDiv, spec, embedOpt)\n",
              "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
              "    }\n",
              "\n",
              "    if(typeof define === \"function\" && define.amd) {\n",
              "      requirejs.config({paths});\n",
              "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
              "    } else {\n",
              "      maybeLoadScript(\"vega\", \"5\")\n",
              "        .then(() => maybeLoadScript(\"vega-lite\", \"5.17.0\"))\n",
              "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
              "        .catch(showError)\n",
              "        .then(() => displayChart(vegaEmbed));\n",
              "    }\n",
              "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300, \"discreteHeight\": 60, \"discreteWidth\": 400}, \"header\": {\"title\": null}, \"mark\": {\"tooltip\": null}, \"title\": {\"anchor\": \"middle\"}}, \"vconcat\": [{\"mark\": {\"type\": \"bar\", \"clip\": true, \"height\": 15}, \"encoding\": {\"color\": {\"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-10, 0, 10], \"interpolate\": \"lab\", \"range\": [\"red\", \"#bbbbbb\", \"green\"]}, \"title\": \"Match weight\", \"type\": \"quantitative\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"title\": \"Comparison name\", \"type\": \"nominal\"}, {\"field\": \"probability_two_random_records_match\", \"format\": \".4f\", \"title\": \"Probability two random records match\", \"type\": \"nominal\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Equivalent match weight\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"domain\": false, \"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}, \"labels\": false, \"ticks\": false, \"title\": \"\"}, \"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-9, 9]}, \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": \"Prior (starting) match weight\", \"titleAlign\": \"right\", \"titleAngle\": 0, \"titleFontWeight\": \"normal\"}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": 20, \"transform\": [{\"filter\": \"(datum.comparison_name == 'probability_two_random_records_match')\"}]}, {\"mark\": {\"type\": \"bar\", \"clip\": true}, \"encoding\": {\"color\": {\"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-10, 0, 10], \"interpolate\": \"lab\", \"range\": [\"red\", \"#bbbbbb\", \"green\"]}, \"title\": \"Match weight\", \"type\": \"quantitative\"}, \"row\": {\"field\": \"comparison_name\", \"header\": {\"labelAlign\": \"left\", \"labelAnchor\": \"middle\", \"labelAngle\": 0}, \"sort\": {\"field\": \"comparison_sort_order\"}, \"type\": \"nominal\"}, \"tooltip\": [{\"field\": \"comparison_name\", \"title\": \"Comparison name\", \"type\": \"nominal\"}, {\"field\": \"label_for_charts\", \"title\": \"Label\", \"type\": \"ordinal\"}, {\"field\": \"sql_condition\", \"title\": \"SQL condition\", \"type\": \"nominal\"}, {\"field\": \"m_probability\", \"format\": \".4f\", \"title\": \"M probability\", \"type\": \"quantitative\"}, {\"field\": \"u_probability\", \"format\": \".4f\", \"title\": \"U probability\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor\", \"format\": \",.4f\", \"title\": \"Bayes factor = m/u\", \"type\": \"quantitative\"}, {\"field\": \"log2_bayes_factor\", \"format\": \",.4f\", \"title\": \"Match weight = log2(m/u)\", \"type\": \"quantitative\"}, {\"field\": \"bayes_factor_description\", \"title\": \"Match weight description\", \"type\": \"nominal\"}], \"x\": {\"axis\": {\"gridColor\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": \"#aaa\"}, \"value\": \"#ddd\"}, \"gridDash\": {\"condition\": {\"test\": \"abs(datum.value / 10) == 1\", \"value\": [3]}, \"value\": null}, \"gridWidth\": {\"condition\": {\"test\": \"abs(datum.value / 10)  <= 1 & datum.value % 10 === 0\", \"value\": 2}, \"value\": 1}, \"title\": \"Comparison level match weight = log2(m/u)\"}, \"field\": \"log2_bayes_factor\", \"scale\": {\"domain\": [-9, 9]}, \"type\": \"quantitative\"}, \"y\": {\"axis\": {\"title\": null}, \"field\": \"label_for_charts\", \"sort\": {\"field\": \"comparison_vector_value\", \"order\": \"descending\"}, \"type\": \"nominal\"}}, \"height\": {\"step\": 12}, \"resolve\": {\"axis\": {\"y\": \"independent\"}, \"scale\": {\"y\": \"independent\"}}, \"transform\": [{\"filter\": \"(datum.comparison_name != 'probability_two_random_records_match')\"}]}], \"data\": {\"name\": \"data-3ac41d0cb87094778550dc4f5831a54f\"}, \"params\": [{\"name\": \"mouse_zoom\", \"select\": {\"type\": \"interval\", \"encodings\": [\"x\"]}, \"bind\": \"scales\", \"views\": []}], \"resolve\": {\"axis\": {\"y\": \"independent\"}, \"scale\": {\"y\": \"independent\"}}, \"title\": {\"text\": \"Model parameters (components of final match weight)\", \"subtitle\": \"Use mousewheel to zoom\"}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.9.3.json\", \"datasets\": {\"data-3ac41d0cb87094778550dc4f5831a54f\": [{\"comparison_name\": \"probability_two_random_records_match\", \"sql_condition\": null, \"label_for_charts\": \"\", \"m_probability\": null, \"u_probability\": null, \"m_probability_description\": null, \"u_probability_description\": null, \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": null, \"is_null_level\": false, \"bayes_factor\": 0.0033430420247643373, \"log2_bayes_factor\": -8.224622793739668, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 0, \"bayes_factor_description\": \"The probability that two random records drawn at random match is 0.003 or one in  300.1 records.This is equivalent to a starting match weight of -8.225.\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": -1}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"\\\"first_name_l\\\" = \\\"first_name_r\\\"\", \"label_for_charts\": \"Exact match on first_name\", \"m_probability\": 0.4953998584571833, \"u_probability\": 0.0057935713975033705, \"m_probability_description\": \"Amongst matching record comparisons, 49.54% of records are in the exact match on first_name comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.58% of records are in the exact match on first_name comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"first_name\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 85.50854463805632, \"log2_bayes_factor\": 6.417996686710603, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on first_name` then comparison is 85.51 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.92\", \"m_probability\": 0.16772823779193205, \"u_probability\": 0.0023429457903817435, \"m_probability_description\": \"Amongst matching record comparisons, 16.77% of records are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.23% of records are in the jaro-winkler distance of first_name >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 71.58861228479536, \"log2_bayes_factor\": 6.1616582088171565, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.92` then comparison is 71.59 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.88\", \"m_probability\": 0.050955414012738856, \"u_probability\": 0.0015484319951285285, \"m_probability_description\": \"Amongst matching record comparisons, 5.10% of records are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.15% of records are in the jaro-winkler distance of first_name >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 32.90775066199099, \"log2_bayes_factor\": 5.040355512425451, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.88` then comparison is 32.91 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"jaro_winkler_similarity(\\\"first_name_l\\\", \\\"first_name_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of first_name >= 0.7\", \"m_probability\": 0.08138711960368011, \"u_probability\": 0.018934945558406913, \"m_probability_description\": \"Amongst matching record comparisons, 8.14% of records are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.89% of records are in the jaro-winkler distance of first_name >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 4.298249464337388, \"log2_bayes_factor\": 2.1037492171812744, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of first_name >= 0.7` then comparison is 4.30 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 0}, {\"comparison_name\": \"first_name\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.20452937013446568, \"u_probability\": 0.9713801052585794, \"m_probability_description\": \"Amongst matching record comparisons, 20.45% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 97.14% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.2105554448019299, \"log2_bayes_factor\": -2.2477279119133526, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  4.75 times less likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 0}, {\"comparison_name\": \"surname\", \"sql_condition\": \"\\\"surname_l\\\" = \\\"surname_r\\\"\", \"label_for_charts\": \"Exact match on surname\", \"m_probability\": 0.5371554671833135, \"u_probability\": 0.004889975550122249, \"m_probability_description\": \"Amongst matching record comparisons, 53.72% of records are in the exact match on surname comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.49% of records are in the exact match on surname comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"surname\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 109.84829303898762, \"log2_bayes_factor\": 6.779368641299035, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on surname` then comparison is 109.85 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.92\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.92\", \"m_probability\": 0.22148371698432145, \"u_probability\": 0.00283905173880724, \"m_probability_description\": \"Amongst matching record comparisons, 22.15% of records are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.28% of records are in the jaro-winkler distance of surname >= 0.92 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 78.01327251519993, \"log2_bayes_factor\": 6.285647687617522, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.92` then comparison is 78.01 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.88\", \"m_probability\": 0.03853254515127222, \"u_probability\": 0.0011314412292407403, \"m_probability_description\": \"Amongst matching record comparisons, 3.85% of records are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.11% of records are in the jaro-winkler distance of surname >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 34.05616142972772, \"log2_bayes_factor\": 5.089843928659545, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.88` then comparison is 34.06 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"jaro_winkler_similarity(\\\"surname_l\\\", \\\"surname_r\\\") >= 0.7\", \"label_for_charts\": \"Jaro-Winkler distance of surname >= 0.7\", \"m_probability\": 0.0892004234559973, \"u_probability\": 0.014729633311540402, \"m_probability_description\": \"Amongst matching record comparisons, 8.92% of records are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 1.47% of records are in the jaro-winkler distance of surname >= 0.7 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 6.055848205407148, \"log2_bayes_factor\": 2.598329043492248, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of surname >= 0.7` then comparison is 6.06 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 1}, {\"comparison_name\": \"surname\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.11362784722509553, \"u_probability\": 0.9764098981702893, \"m_probability_description\": \"Amongst matching record comparisons, 11.36% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 97.64% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.11637310051651938, \"log2_bayes_factor\": -3.1031704750969547, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  8.59 times less likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 1}, {\"comparison_name\": \"dob\", \"sql_condition\": \"\\\"dob_l\\\" = \\\"dob_r\\\"\", \"label_for_charts\": \"Exact match on dob\", \"m_probability\": 0.4074598725152314, \"u_probability\": 0.0017477477477477479, \"m_probability_description\": \"Amongst matching record comparisons, 40.75% of records are in the exact match on dob comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.17% of records are in the exact match on dob comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 233.13425695459117, \"log2_bayes_factor\": 7.8650172007075945, \"comparison_vector_value\": 5, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `exact match on dob` then comparison is 233.13 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"damerau_levenshtein(\\\"dob_l\\\", \\\"dob_r\\\") <= 1\", \"label_for_charts\": \"Levenshtein distance of dob <= 1\", \"m_probability\": 0.10858423514239293, \"u_probability\": 0.0016436436436436436, \"m_probability_description\": \"Amongst matching record comparisons, 10.86% of records are in the levenshtein distance of dob <= 1 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.16% of records are in the levenshtein distance of dob <= 1 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 66.06312479126099, \"log2_bayes_factor\": 6.045773305719355, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `levenshtein distance of dob <= 1` then comparison is 66.06 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 2629800.0\", \"label_for_charts\": \"Abs difference of 'transformed dob <= 1 month'\", \"m_probability\": 0.08103908705052423, \"u_probability\": 0.0024084084084084086, \"m_probability_description\": \"Amongst matching record comparisons, 8.10% of records are in the abs difference of 'transformed dob <= 1 month' comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.24% of records are in the abs difference of 'transformed dob <= 1 month' comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 33.648398987312426, \"log2_bayes_factor\": 5.072465956988507, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `abs difference of 'transformed dob <= 1 month'` then comparison is 33.65 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 31557600.0\", \"label_for_charts\": \"Abs difference of 'transformed dob <= 1 year'\", \"m_probability\": 0.1254745174675755, \"u_probability\": 0.033053053053053054, \"m_probability_description\": \"Amongst matching record comparisons, 12.55% of records are in the abs difference of 'transformed dob <= 1 year' comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 3.31% of records are in the abs difference of 'transformed dob <= 1 year' comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 3.796155146883947, \"log2_bayes_factor\": 1.9245389557589478, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `abs difference of 'transformed dob <= 1 year'` then comparison is 3.80 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ABS(EPOCH(try_strptime(\\\"dob_l\\\", '%Y-%m-%d')) - EPOCH(try_strptime(\\\"dob_r\\\", '%Y-%m-%d'))) <= 315576000.0\", \"label_for_charts\": \"Abs difference of 'transformed dob <= 10 year'\", \"m_probability\": 0.2316662350269591, \"u_probability\": 0.30747947947947946, \"m_probability_description\": \"Amongst matching record comparisons, 23.17% of records are in the abs difference of 'transformed dob <= 10 year' comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 30.75% of records are in the abs difference of 'transformed dob <= 10 year' comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.7534364095423156, \"log2_bayes_factor\": -0.4084423422007769, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `abs difference of 'transformed dob <= 10 year'` then comparison is  1.33 times less likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 2}, {\"comparison_name\": \"dob\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.04577605279731682, \"u_probability\": 0.6536676676676677, \"m_probability_description\": \"Amongst matching record comparisons, 4.58% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 65.37% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.07002955027689989, \"log2_bayes_factor\": -3.8358923671173697, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 5, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  14.28 times less likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 2}, {\"comparison_name\": \"city\", \"sql_condition\": \"\\\"city_l\\\" = \\\"city_r\\\"\", \"label_for_charts\": \"Exact match on city\", \"m_probability\": 0.5644811598609882, \"u_probability\": 0.0551475711801453, \"m_probability_description\": \"Amongst matching record comparisons, 56.45% of records are in the exact match on city comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 5.51% of records are in the exact match on city comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"city\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 10.235829933779886, \"log2_bayes_factor\": 3.355556177493555, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `exact match on city` then comparison is 10.24 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 3}, {\"comparison_name\": \"city\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.4355188401390118, \"u_probability\": 0.9448524288198547, \"m_probability_description\": \"Amongst matching record comparisons, 43.55% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 94.49% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.4609384776445844, \"log2_bayes_factor\": -1.117353890697282, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 1, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  2.17 times less likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 3}, {\"comparison_name\": \"email\", \"sql_condition\": \"\\\"email_l\\\" = \\\"email_r\\\"\", \"label_for_charts\": \"Exact match on email\", \"m_probability\": 0.5523957087606948, \"u_probability\": 0.0021938713143283602, \"m_probability_description\": \"Amongst matching record comparisons, 55.24% of records are in the exact match on email comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.22% of records are in the exact match on email comparison level\", \"has_tf_adjustments\": true, \"tf_adjustment_column\": \"\\\"email\\\"\", \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 251.79038768270107, \"log2_bayes_factor\": 7.976079397743489, \"comparison_vector_value\": 4, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on email` then comparison is 251.79 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), '') = NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')\", \"label_for_charts\": \"Exact match on username\", \"m_probability\": 0.22216648099192782, \"u_probability\": 0.0010390328952024346, \"m_probability_description\": \"Amongst matching record comparisons, 22.22% of records are in the exact match on username comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.10% of records are in the exact match on username comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 213.82044978339513, \"log2_bayes_factor\": 7.740256028755592, \"comparison_vector_value\": 3, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `exact match on username` then comparison is 213.82 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"jaro_winkler_similarity(\\\"email_l\\\", \\\"email_r\\\") >= 0.88\", \"label_for_charts\": \"Jaro-Winkler distance of email >= 0.88\", \"m_probability\": 0.20620379651952536, \"u_probability\": 0.0009135769109519858, \"m_probability_description\": \"Amongst matching record comparisons, 20.62% of records are in the jaro-winkler distance of email >= 0.88 comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.09% of records are in the jaro-winkler distance of email >= 0.88 comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 225.71038524239003, \"log2_bayes_factor\": 7.81832899019637, \"comparison_vector_value\": 2, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler distance of email >= 0.88` then comparison is 225.71 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"jaro_winkler_similarity(NULLIF(regexp_extract(\\\"email_l\\\", '^[^@]+', 0), ''), NULLIF(regexp_extract(\\\"email_r\\\", '^[^@]+', 0), '')) >= 0.88\", \"label_for_charts\": \"Jaro-Winkler >0.88 on username\", \"m_probability\": 0.010649001646986053, \"u_probability\": 0.000501823937001795, \"m_probability_description\": \"Amongst matching record comparisons, 1.06% of records are in the jaro-winkler >0.88 on username comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 0.05% of records are in the jaro-winkler >0.88 on username comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 21.220593243538243, \"log2_bayes_factor\": 4.407393083729446, \"comparison_vector_value\": 1, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `jaro-winkler >0.88 on username` then comparison is 21.22 times more likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 4}, {\"comparison_name\": \"email\", \"sql_condition\": \"ELSE\", \"label_for_charts\": \"All other comparisons\", \"m_probability\": 0.008585012080866098, \"u_probability\": 0.9953516949425154, \"m_probability_description\": \"Amongst matching record comparisons, 0.86% of records are in the all other comparisons comparison level\", \"u_probability_description\": \"Amongst non-matching record comparisons, 99.54% of records are in the all other comparisons comparison level\", \"has_tf_adjustments\": false, \"tf_adjustment_column\": null, \"tf_adjustment_weight\": 1.0, \"is_null_level\": false, \"bayes_factor\": 0.008625104196323199, \"log2_bayes_factor\": -6.8572423991754246, \"comparison_vector_value\": 0, \"max_comparison_vector_value\": 4, \"bayes_factor_description\": \"If comparison level is `all other comparisons` then comparison is  115.94 times less likely to be a match\", \"probability_two_random_records_match\": 0.0033319033319033323, \"comparison_sort_order\": 4}]}}, {\"mode\": \"vega-lite\"});\n",
              "</script>"
            ],
            "text/plain": [
              "alt.VConcatChart(...)"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "linker.visualisations.match_weights_chart()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "state": {
          "54186cece08b4f6fa03f33cc282f36a6": {
            "model_module": "@jupyter-widgets/controls",
            "model_module_version": "2.0.0",
            "model_name": "FloatProgressModel",
            "state": {
              "_dom_classes": [],
              "_model_module": "@jupyter-widgets/controls",
              "_model_module_version": "2.0.0",
              "_model_name": "FloatProgressModel",
              "_view_count": null,
              "_view_module": "@jupyter-widgets/controls",
              "_view_module_version": "2.0.0",
              "_view_name": "ProgressView",
              "bar_style": "",
              "description": "",
              "description_allow_html": false,
              "layout": "IPY_MODEL_8863bef4905c44fc9705add5d5165a71",
              "max": 100,
              "min": 0,
              "orientation": "horizontal",
              "style": "IPY_MODEL_ce7ee37dfbeb4d26ae9171f7f3b857e7",
              "tabbable": null,
              "tooltip": null,
              "value": 100
            }
          },
          "8863bef4905c44fc9705add5d5165a71": {
            "model_module": "@jupyter-widgets/base",
            "model_module_version": "2.0.0",
            "model_name": "LayoutModel",
            "state": {
              "_model_module": "@jupyter-widgets/base",
              "_model_module_version": "2.0.0",
              "_model_name": "LayoutModel",
              "_view_count": null,
              "_view_module": "@jupyter-widgets/base",
              "_view_module_version": "2.0.0",
              "_view_name": "LayoutView",
              "align_content": null,
              "align_items": null,
              "align_self": null,
              "border_bottom": null,
              "border_left": null,
              "border_right": null,
              "border_top": null,
              "bottom": null,
              "display": null,
              "flex": null,
              "flex_flow": null,
              "grid_area": null,
              "grid_auto_columns": null,
              "grid_auto_flow": null,
              "grid_auto_rows": null,
              "grid_column": null,
              "grid_gap": null,
              "grid_row": null,
              "grid_template_areas": null,
              "grid_template_columns": null,
              "grid_template_rows": null,
              "height": null,
              "justify_content": null,
              "justify_items": null,
              "left": null,
              "margin": null,
              "max_height": null,
              "max_width": null,
              "min_height": null,
              "min_width": null,
              "object_fit": null,
              "object_position": null,
              "order": null,
              "overflow": null,
              "padding": null,
              "right": null,
              "top": null,
              "visibility": null,
              "width": "auto"
            }
          },
          "ce7ee37dfbeb4d26ae9171f7f3b857e7": {
            "model_module": "@jupyter-widgets/controls",
            "model_module_version": "2.0.0",
            "model_name": "ProgressStyleModel",
            "state": {
              "_model_module": "@jupyter-widgets/controls",
              "_model_module_version": "2.0.0",
              "_model_name": "ProgressStyleModel",
              "_view_count": null,
              "_view_module": "@jupyter-widgets/base",
              "_view_module_version": "2.0.0",
              "_view_name": "StyleView",
              "bar_color": "black",
              "description_width": ""
            }
          }
        },
        "version_major": 2,
        "version_minor": 0
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}